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ABSTRACT 

Expressing concern for the quality of care supported by Child 
Care and Development Fund (CCDF) funds, welfare reform legislation required 
states to set aside at least 4 percent of their total grant to improve child 
care quality and availability. In preparation for reauthorizing CCDF, this 
General Accounting Office report was commissioned to examine quality 
improvements undertaken by states with their 4 percent set-asides and 
evidence for effectiveness of these initiatives. Data collection methods 
included: (1) surveys of CCDF lead state agencies in the 50 states and the 

District of Columbia; (2) case studies conducted in California, 

Massachusetts, South Dakota, Tennessee, and Wisconsin to amplify survey 
information; and (3) reports of evaluations of state initiatives. Findings 
revealed that states undertook a variety of quality initiatives, such as 
training caregivers and. raising caregiver compensation. Most states spent 
more than the- 4 percent set-aside. Factors influencing state initiatives 
included the perspective of the . governor or state legislature about high 
quality care, recent events in the child care community, and previous’ 
research. Few states have evaluated the effects of quality improvement 
initiatives on children's development . A few studies of child care quality 
showed that some of the quality attributes addressed by state initiatives are 
linked to children's development. The report recommends that the Department 
of Health and Human Services include selected state quality improvement 
initiatives in a major impact evaluation of state child care subsidy 
strategies. The report's five appendices include information on the study's 
methodology, an outline of state-initiated studies of quality improvement, 
and findings from two research reviews reporting effects of program quality 
and child-caregiver interactions on children's development. A 47-item 
bibliography concludes the report. (KB) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. . 



ED 469 418 



United States General Accomiting Office 



ED 469 418 

S3 


Report to Congressional Requesters 


September 2002 


CHILD CARE 




States Have 
Undertaken a Variety 
of Quality 
Improvement 
Initiatives, but More 
Evaluations of 
Effectiveness Are 
Needed 

U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
ly CENTER (ERIC) 

M This document has been reproduced as 
A \ received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 


QQi 


• Points of view or opinions stated in this 
document do not necessarily represent 
official OER) position or policy. 


Wi 

P=3 


GAO 


GAO-02-897 

O 

ERIC 


o ElpT COPY AVAIILABLE 



Contents 


Letter 




1 




Results in Brief 


3 




Background 

States Undertook a Variety of Initiatives, Primarily Using the 4 


4 




Percent Set-Aside 

Few States Have Evaluated the Effectiveness of State Quality 


11 




Improvement Initiatives 


24 




Conclusion 


30 




Recommendation 


30 




Agency Comments 


30 


Appendix I 


Scope and Methodology 


32 




Scope 


32 




Methodology 


32 


Appendix II 


State-Initiated Studies of Quality Improvement 


40 


Appendix III 


Child Care Quality Research Findings 


45 




Child Care Quality Linked to Socio-Emotional Development 


46 




Child Care Quality Linked to Cognitive Development 


47 




Child Care Quality Linked to Child Development Over Time 


48 


Appendix IV 


Comments from the Department of Health and Human 
Services 


51 



Appendix V 


GAO Contacts and Staff Acknowledgments 


54 




GAO Contacts 


54 




Staff Acknowledgments 


54 


Bibliography 




55 



Related GAO Products 60 



GAO-02*897 Child Care Quality Improvement Initiatives 



o 



Page i 



Tables 



Table 1: Types and Descriptions of Child Care Providers 4 

Table 2: Rules for Obligating and Spending Funds in the CCDF 

Funding Streams 6 

Table 3: Categories Used to Describe States’ Child Care Quality 

Improvement Initiatives 9 

Table 4: States’ Reported CCDF Quality Improvement 

Expenditures in Fiscal Year 2000 19 

Table 5: States’ Reported Expenditures Devoted to Each Provider 

Type, by Initiative 22 

Table 6: Comparison of Quality Improvement Expenditures 

Distributed to Individual Providers that Were Devoted to 
Each Provider Type, with Percentage of CCDF-Subsidized 
Children, by State 23 

Table 7: Major Reviewers’ Findings Regarding Child Care Quality 

Research 28 

Table 8: Data Quality Criteria 37 

Table 9: Criteria for Assessing Evaluations 37 

Table 10: State-initiated Studies of Quality Improvement 40 



Figures 

Figure 1: State Expenditures for Quality Improvement Initiatives in 



Fiscal Year 2000 3 

Figure 2: Percentage of States that Reported Undertaking Nine 

Categories of Initiatives 12 

Figure 3: States’ Reported Expenditures for Each Initiative, Fiscal 

Year 2000 13 

Figure 4: States’ Reported Expenditures from Each Funding 

Source, Fiscal Year 2000 15 

Figure 5: Massachusetts’s Expenditures on Quality Improvement 

Initiatives, Fiscal Year 2000 16 

Figure 6: Tennessee’s Fiscal Year 2000 Expenditures on Quality 

Improvement Initiatives 17 

Figure 7: South Dakota’s Fiscal Year 2000 Expenditures on Quality 

Improvement Initiatives 18 

Figure 8: States’ Reported Expenditures on Quality Improvement 

Initiatives Targeted to Providers 22 




GAO-02-897 Child Care Quality Improvement Initiatives 



Page ii 



4 



Abbreviations 



CCDBG 

CCDF 

ECERS 

FDCRS 

HHS 

ITERS 

MCHB 

MOE 

NAEYC 

NICHD 

OPRE 

PRWORA 

TANF 

TEACH 



Page iii 



Child Care and Development Block Grant Act of 1990 

Child Care and Development 

Early Care Environment Rating Scale 

Fairly Day Care Rating Scale 

Department of Health and Human Services 

Infant/Toddler Enviromnent Rating Scale 

Maternal and Child Health Bureau 

maintenance of effort 

National Association for the Education of Young 
children 

National Institute of Child Health and Human Development 
Office of Planning, Research and Evaluation 
Person^ Responsibility and Work Opportunity 
Reconciliation Act of 1996 
Temporary Assistance for Needy Families 
Teacher Education and Compensation Helps 



GAO-02-897 Child Care Quality Improvement Initiatives 



i 

^ G A O 

^^J^^^PAcTOgntablllty * Integrity * Reliability 

United States General Accounting Office 
Washington, DC 20548 



September 6, 2002 

The Honorable Edward M. Kennedy 
Chairman, Committee on Health, Education, Labor 
and Pensions 
United States Senate 

The Honorable Christopher J. Dodd 

Chairman, Subcommittee on Children and Families 

Committee on Health, Education, Labor and Pensions, 

United States Senate 

The Honorable Jack Reed 
United States Senate 

The demand for child care has increased dramatically in the past several 
decades as the number of mothers who work outside the home has grown. 
Welfare reform has further increased this demand. To support low-income 
parents moving into the workforce, welfare reform established the Child 
Care and Development Fund (CCDF). In fiscal year 2000, states spent $5.3 
billion in federal CCDF to subsidize child care for low-income families. 

Out of concern for the quality of care supported by CCDF fimds, welfare 
reform legislation also required states to set aside at least 4 percent of the 
total grant to improve the quality and availability of child care. Department 
of Health and Human Services (HHS) regulations provide examples of 
allowable activities, such as providing child care providers with financial 
incentives for meeting state and local standards, improving the 
compensation of child care staff, and offering resource and referral 
services. However, the regulations do not limit states’ use of funds to these 
activities; rather, the fimd’s block grant structure allows states 
considerable flexibility in choosing appropriate quality and availability 
improvements to pursue. 

As Congress considers the CCDF’s structure and funding level in 
preparation for reauthorization in 2002, interest has increased in the types 
of quality improvement initiatives 4 percent set-aside funds are supporting, 
the estimated percentage of federal and state fimds being spent on such 
initiatives, and the extent to which states are assessing the initiatives’ 
effects. Accordingly, in preparation for CCDF’s reauthorization, you asked 
us to examine (1) what quality improvement initiatives states have 
undertaken wi^ the 4 percent set-aside and other funding sources and (2) 
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what evidence has been gathered, if any, about the effectiveness of states’ 
initiatives. 

To determine what initiatives states have conducted, we surveyed CCDF 
lead state agencies in the 50 states and the District of Columbia about the 
use of CCDF and other funds in fiscal year 2000. We received responses 
from 42 states. We asked them to classify their quality improvement 
initiatives into nine general categories, which include the major activities 
identified in the law, HHS’s regulations, and in the child care literature and 
to identify the funding sources for each initiative category and the amount 
spent. We also conducted case studies of five states — California, 
Massachusetts, South Dakota, Tennessee, and Wisconsin — to gather data 
that would amplify information on states’ initiatives collected by the 
survey. We selected states that were diverse geographically and in 
population density and that represented a variety of child care quality 
improvement initiatives. We also considered the state’s income 
distribution, licensing caseloads, use of Temporary Assistance for Needy 
Families funds and whether state licensing requirements refiected child-to- 
staff ratios recommended by national child care accrediting bodies. To 
examine the evidence of effectiveness, we asked state lead agencies for 
evaluations of their initiatives, contacted HHS and researchers regarding 
their work, and assessed the evaluations we identified. We also reviewed 
major summaries and methodological critiques of the research literature 
on child care quality. Appendix I provides additional details about our 
scope and methodology. We conducted our work between December 2001 
and June 2002 in accordance with generally accepted government auditing 
standards. 
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Results in Brief 



Using primarily the 4 percent quality set-aside, states reported undertaking 
a variety of child care quality improvement initiatives, such as training 
caregivers, raising the compensation of caregivers, referring parents to 
child care providers, and efforts to enhance the safety of child care 
facilities, as shown in the figure below. State officials in the five case study 
states cited several factors that influenced the initiatives states undertook, 
including the perspective of the governor or state legislature about high 
quality care, recent events in the child care community and previous 
research. 



Figure 1: State Expenditures for Quality Improvement Initiatives in Fiscal Year 2000 




Source: GAO analysis of GAO survey data. 



The majority of states reported expenditures exceeding the 4 percent set- 
aside’s minimum requirement. Among the 34 states that tracked the type of 
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provider targeted, child care centers received over two-thirds of 
expenditures on quality initiatives that distributed funds and resources to 
providers, while less than a third of such expenditvures went to family child 
care or after-school care. 

While few states have evaluated the effects of their quality improvement 
initiatives on children’s development, some studies provide useful findings 
about them. Officials in four of five states we talked to explained that 
states must make trade-offs between serving more families and conducting 
evaluations of their own quality improvement initiatives. Out of the 
handful of state-sponsored studies, a few had study designs that isolated 
an initiative’s effect and survey response rates that provided reliable 
estimates. The research on child care quality does not evaluate initiatives 
as actually implemented by states, but a few studies, using rigorous 
methods, show that some of the attributes of child care quality that these 
initiatives address, such as caregiver qualifications, are linked to children’s 
social, emotional and cognitive development. To provide states with 
rigorous research evidence about how to modify ongoing initiatives or 
invest in new ones, we are recommending that HHS include selected state 
quality improvement initiatives in a mgjor impact evaluation of state child 
care subsidy strategies. 



Background 



Child care services are supplied by providers operating in varied settings: 
in center care, a child is cared for in a nonresidential setting and in family 
child care, a child is cared for in the home of a provider. CMd care centers 
provide care outside of the home, but family child care is provided to a 
small number of unrelated children — typically fewer than six — ^in a 
provider’s home. Some child care centers and family child care homes also 
offer school-aged care for children before and after school. (See table 1.) 
Generally, children in center-based care and family child care have not yet 
started school and after-school care is offered to children in kindergarten 
through age 12. 



Table 1 : Types and Descriptions of Child Care Providers 


Type of provider 


Description* 


Child care center 


Care typically provided for 12 or more children in a nonresidential 
facility. 


Family child care 


Care provided for a small group of children in a provider’s home. 


informal care 


Legally operating care given by adults, including relatives and 
friends and usually unregulated. 



Table 1 provides a general description of different types of child care providers. In actuality, states 
define child care differently and have different licensure and regulatory requirements. 
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Source: U.S. General Accounting Office, States Increased Spending on Low-Income Families, 
GA(D-0l-2i)3 (Washington, D.C.: Feb. 2, 2002) and Implications of Increased Work Participation (or 
Child Care, GAO/lVEHS-95-75 (Washington, D.C.: May 29, 1997). 



Research on child care quality identified two broad sets of attributes that 
pertain to quality in all child care settings: structural attributes of the child 
care environment and children’s daily interactions with their caregivers. 
Structural attributes of child care include characteristics such as child-to- 
staff ratios, the number of children per caregiver in a classroom; group 
size, the number of children assigned to a team of caregivers in a 
classroom; caregiver formal education; caregivers’ specialized training; 
caregiver wages; staff turnover; the amount of floor space per child; and 
health and safety features, such as frequent staff and child hand washing. 
Child-caregiver interactions refers to actual experiences that occur in 
child care settings, and include such attributes as caregiver sensitivity and 
responsiveness, caregiver participation in children’s play and learning 
activities, and language stimulation by caregivers. 

State and local governments are responsible for the oversight of child care 
providers that operate in their state. Each state establishes its own child 
care standards, determining the areas and types of providers that the 
standards will cover and the specific criteria that will be used to determine 
provider compliance. Most child care providers are required to meet a 
state’s standards to obtain a license to operate legally in a state. State child 
care standards primarily focus on the structmal attributes of care. 

States can turn to organizations such as the National Association for the 
Education of Young Children (NAEYC) and the Maternal and Child Health 
Bvu-eau (MCHB) in HHS’s Public Health Service that have developed 
standards based on research and professional practice. NAEYC, the 
nation’s largest association of early child professionals, was formed to 
improve professional practice in early childhood care and education and 
increase public understanding of high quality early childhood programs. 
NAEYC also accredits, through a voluntary system, early childhood 
centers and schools. In 1998, we reported that state licensing standards 
varied in the extent to which the standards reflected those of NAEYC and 
MCHB. For example, we found that only two states had standards for 
caregiver education and training that matched NAEYC standards. 
Typically, state standards tended to require significantly fewer years of 
education than the standards set by NAEYC. Thus, to achieve 
accreditation by a national accrediting body, child care providers may 
have to meet higher standards than those they would meet to obtain and 
keep a state operating license. 
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CCDF Structure and 
Spending Requirements 



Title I of the Personal Responsibility and Work Opportunity Reconciliation 
Act of 1996 (PRWORA) overhauled the nation’s welfare system by 
replacing the legal entitlement to cash assistance under the previous 
welfare program with the Temporary Assistance for Needy Families 
(TANK) block grant. Title VI of PRWORA amended the Child Care and 
Development Block Grant Act of 1990 (CCDBG) and combined CCDBG 
funds with the funding of three other federal child care programs. HHS 
named the combined set of funds the CCDF. 

Each state receives an annual CCDF allocation composed of funds from 
three separate funding streams: discretionary, mandatory and matching. 
Assessing the portion of CCDF funds states spend on quality improvement 
is complicated to some extent by the distinct set of rules covering esich 
stream that determine the time period allowed for obligating and spending 
the funds. (See table 2.) 



Table 2: Rules for Obligating and Spending Funds in the CCDF Funding Streams 


Funding streams 


Time period for obligating 
funds 


Time period for spending funds 


Discretionary 


Within 2 fiscal years after a 
grant award 


Within 3 years after a grant award 


Mandatory 


To receive CCDF matching 
funds, within the fiscal year 
of a grant award 


Available until spent 


Matching 


Within the fiscal year of a 
grant award 


Within 2 years after a grant award 



Each state receives a share of the total amount of money in the 
discretionary funding stream, which is determined each year by the 
congressional appropriations process. A state’s share of discretionary 
funds is based on a formula stipulated in the statute. A state must obligate 
discretionary funds within 2 fiscal years after a grant award and spend the 
funds by the end of the following fiscal year. 

A state’s share of mandatory funds is based on the amount of funds the 
state received from a set of federal child care programs in a base year. 
Mandatory funds are available until they are spent. However, to receive 
federal matching funds, a state must obligate all mandatory funds by the 
end of the fiscal year in which they were awarded; maintain program 
spending of state funds at a specified level, referred to as a state’s 
maintenance of effort (MOE); and spend additional state funds above that 
level. States may spend more of their own funds on child care than the 
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amount actually accounted for under CCDF’s MOE and matching 
requirements. Federal and state matching funds must be committed by the 
end of the fiscal year in which they are received and spent by the end of 
the following fiscal year. 

Finally, funds transferred from the TANF block grant represent an 
additional source of funds for the CCDF. PRWORA allowed states the 
flexibility to transfer up to 30 percent of TANF funds to the CCDF. 
Transferred TANF funds are treated as part of the discretionary funding 
stream and are subject to CCDF rules. 

States must spend at least 4 percent of their CCDF funds — of 
discretionary, mandatory and matching, but not of state MOE funds — for a 
given fiscal year on activities intended to improve the quality and 
availability of child care. Specifically, the law requires that states use at 
least 4 percent of these funds for activities to provide comprehensive 
consumer education to parents and the public, activities that increase 
parental choice, and activities designed to improve the quality and 
availability of child care. As stated earlier, HHS, through its regulations, 
has provided illustrative examples of activities designed to improve the 
quality of child care. In addition, the regulations permit other expenditures 
that are consistent with the intent of the regulation. This provision of 
PRWORA is known as the 4 percent set-aside. Congress also has 
earmarked money in CCDFs discretionary fund for resource and referral 
services and school-age care, infant and toddler care and quality-related 
activities. Any funds expended for the activity beyond the designated 
earmarks can be used to meet the 4 percent set-aside requirement. 
Earmarked funds must be tracked and reported separately from 4 percent 
set-aside expenditures. For fiscal year 2001, Congress provided 
$19,100,000 for the resource and referral services and school-age care 
earmark, $100,000,000 for the infant and toddler earmark, and $172,600,000 
for the quality-related activities earmark.' These earmark amounts were 
continued for fiscal year 2002. HHS guidance for expenditure of the 
quality-related activities earmark includes activities similar to those 
approved for the 4 percent set-aside, but covers additional suggestions 



‘Congress specified that $1,000,000 of the earmark for resource and referral services and 
school-age care be used for a hotline to be operated by Child Care Aware. Child Care 
Aware is a national toll-free child care consumer telephone hotline and web-site operated 
by the National Association of Child Care Resource and Referral Agencies, through a 
cooperative agreement with the Child Care Bureau in the Department of Health and Human 
Services. 
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such as specific health activities, special needs child care and activities 
that support cultural diversity. 



Methodology 



Framework for Analyzing 
States’ Quality Improvement 
Expenditures 



To encompass the broad range of quality improvement initiatives that 
states are undertaking, including those allowed by the 4 percent set-aside 
provision and HHS’s regulations as well as strategies suggested in child 
care quality research and practice, we developed a framework for 
describing the initiatives and analyzing states’ expenditures on them. To 
assess evidence on the effectiveness of states’ initiatives that has been 
developed in the research community, we developed criteria for data and 
research quality that reflect GAO’s methodological standards and those of 
the broader policy research community. 

The CCDF 4 percent set-aside provision and HHS regulations specify 
several types of activities for which quality improvement funds may be 
expended but also allow states the discretion to include other activities. 
HHS also requires states to report total expenditures of 4 percent set-aside 
funds annually but does not require separate reporting for the quality 
improvement initiatives the states undertake. Thus, to examine states’ 
quality improvement expenditures, we developed a framework to guide 
data collection and analysis. Beginning with examples of the allowable 
activities included in the 4 percent set-aside provision and HHS 
regulations, we specified nine categories to characterize states’ initiatives. 

The categories are based on several sources. (See table 3.) Most 
categories — caregiver compensation, meeting state standards, safety 
equipment or improvement, caregiver education and training and resource 
and referral — are based on examples in the law and regulations. Because 
our analytic framework includes the full range of states’ quality 
improvement initiatives, including those funded by sources other than 
CCDF, we identified additional categories based on child care quality 
literature. On-site training and enhanced inspections were included as 
categories based on the Department of Defense’s child development 
program, which has been widely recognized as a model of high quality 
care.^ Incentives for achieving accreditation or exceeding standards is a 



^Recent studies of the military child care program include Gail L Zellman and Susan M. 
Gates, Examining the Cost of Military Child Care (Santa Monica, Calif.: RAND, 2002), 
httpy/www.rand.org4>ublications/MR/MR1415/ and Gail L. Zellman and Anne S. Johansen, 
Examining the Implementation and Outcomes of the Milit^ Child Care Act of 1989 (Santa 
Monica, Calif.: RAND, 1998), http://www.rand.org/publications/MR/MR666 
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category based on several studies of child care quality improvement 
strategies that look beyond the scope of activities cited in the law. The 
final category for other quality-related activities included initiatives that 
may be unique to a state and those that may foster the availability of high 
quality care, such as strategies that provide consumer education or 
increase parental choice. Because activities to provide comprehensive 
consumer education to parents and the public and increase parental 
choice were not included among the activities noted by the law or HHS 
regulations as designed to improve child care quality, we did not include 
these activities in our framework. However, states were free to report 
these or other quality-related activities when they construed them as such. 



Table 3: Categories Used to Describe States’ Child Care Quality Improvement Initiatives 


Initiative 


Basis 


Description 


Caregiver compensation 


CCDF regulations 


Funding for caregivers or providers to increase caregivers’ salary 
or benefits 


On-site training 


Military child development program 


Funding for training of caregivers provided at employment site 


Meeting State Standards 


CCDF regulations 


Funding for the purpose of helping providers meet state 
standards and consequently become licensed 


Safety equipment or 
improvement 


CCDF regulations 


Funding for the purpose of helping providers improve safety 


Incentives for accreditation 
or exceeding standards 


Literature on child care quality and 
CCDF regulations 


, Funding to encourage providers to meet some higher standard 


Caregiver education or 
training 


CCDF regulations 


Funding for caregivers to receive training or education, often in 
child development or health and safety; may include 
scholarships, funding of class at a community college, or other 
training not at the caregivers’ place of employment 


Resource and Referral 
Activities 


CCDBG Act and CCDF regulations 


Funding for parent and provider support, including activities to 
help parents find a provider, coordination of caregiver training, 
provision of materials and training to caregivers, or provision of 
technical assistance to caregivers 


Enhanced Inspections 


Military child development program 


Funding to increase the frequency of inspections of child care 
providers, increase the scope of the inspections, or decrease 
inspector caseload 


Other quality-related 
activities 


CCDBG Act and CCDF regulations 


Funding for other state-initiated activities 



Source: GAO analysis. 



Methods for Evaluating the Prior to and since CCDF’s creation, a large body of research on child care 

Effectiveness of States’ Quality that included an analysis of its effects on children’s development has been 

Improvement Initiatives accumulating. In IddO, the Hational Research Council assessed this 

research, focusing on the costs, effects and feasibility of child care policies 
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and programs.^ As part of this assessment, the council concluded that child 
care quality is linked to children's development. The council emphasized 
that it woidd be important for future research to examine exactly how the 
various components of quality affected children’s development, what 
magnitude of improvement in development could be expected from 
measured improvements in quality and, most importantly, whether the 
quality of child care has an effect on children’s development that is 
separate from that of family characteristics. 

Noting that studies using random assignment of children to differing child 
care arrangements of varying quality provide the most rigorous evidence 
of whether child care quality has an effect that is separate from family 
characteristics, the council also found that random assignment had been 
used rarely in studies of community-based child care settings. Pointing out 
the contributions of experimental designs, the term given to studies that 
employ random assignment, to research on early interventions for children 
from disadvantaged families, the council urged that experimental designs 
be used in future research on child care quality. Other reviews of the 
research on child care quality, while agreeing on the importance of looking 
at the effects of child care quality separately from the effects of family 
characteristics, acknowledged the practical difficulties of random 
assignment and recommended an ^temative approach that uses advanced 
statistical methods and a comparison group, an approach known as quasi- 
experimental design.^ In conducting our assessment of research on the 
effectiveness of states’ quality improvement initiatives, we also used the 
criterion that to determine a program’s effect, an evaluation should 
employ an experimental or quasi-experimental design. Appendix I 
provides additional details about these designs and our complete scope 
and methodology. 



National Research Council is the principal operating body of the National Academy of 
Sciences, the National Academy of Engineering and the Institute of Medicine. It operates 
under a charter granted by Congress, to advise the government, the public and the 
scientific and engineering communities about scientific and technical matters. The National 
Academies of Science and Engineering are private, nonprofit societies of scholars in the 
fields of science and engineering. The Institute of Medicine is an association of eminent 
members of the professions pertaining to public health who advise on medical, research 
and educational issues. 

^An experimental design requires random assignment of study participants to a group that 
is receiving services and to a control group that is not A quasi-experimental design does 
not require random assignment, but does require statistical controls for factors other than 
the program that may have influenced the outcome. See appendix I for a discussion of 
these research methods and considerations in their use. 
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States Undertook a 
Variety of Initiatives, 
Primarily Using the 
4 Percent Set-Aside 


Using primarily the 4 percent quality set-aside, states reported undertaking 
a variety of child care quality improvement initiatives, such as efforts to 
train caregivers, raise the compensation of caregivers, and enhance the 
safety of child care facilities. State officials in the five case study states 
cited several factors that influenced the initiatives they undertook, 
including state legislators’ perspectives on what constitutes quality care, 
recent events in the state child care community, evaluations, and other 
previous research. While states are required to spend 4 percent of the 
CCDF on quality, the majority of states reported quality expenditures in 
excess of this minimum requirement. Among the 34 states that tracked the 
type of provider targeted, child care centers received over two-thirds of 
those quality expenditures distributed to providers, while less than a third 
of such expenditures went to family child care or after-school care. 
Because initiatives that distributed funds to providers constituted 54 
percent of states’ expenditures for quality improvement, expenditures 
devoted to centers represented about 39 percent of states’ total reported 
expenditures for quality improvement 


Resource and Referral 
Predominates in States’ 
Quality Improvement 
Expenditures 


States reported undertaking resource and referral activities more than any 
other initiative. (See fig. 2.) Resource and referral services are identified in 
the CCDBG Act as an example of activities for which states may make 
expenditures for quality improvement. Two of the states we visited 
described the use of resource and referral agencies to deliver technical 
assistance to providers. In South Dakota, the state’s five resource and 
referral agencies provided child care providers with technical assistance 
needed to meet regulatory requirements. In Massachusetts, the child care 
agency used resource and referral agencies to assist providers in caring for 
children with special needs, such as a child with a disability. In 
collaboration with the state’s Department of Public Health, the 
Massachusetts child care agency developed a consultation program for 
special needs children. Consultation program representatives helped 
resource and referral agencies understand what a child’s needs were when 
placing the child with a provider. Three states described the use of 
resource and referral agencies as a vehicle for training. The South Dakota 
and California child care agencies used resource and referral agencies to 
deliver all training for child care providers. In Massachusetts, the state’s 
resource and referral agencies trained providers in using the Early 
Childhood Environment Rating Scale for self-assessments of quality, in 
using information technology and in early literacy. Providers who then 
implemented early literacy initiatives for their staff were offered rate 
increases. 
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Initiatives received different proportions of total reported expenditures for 
quality improvement, using all funding sources.® In addition to being the 
most frequently undertaken, states reported that resource and referral 
activities received a larger share of reported expenditures on quality 
improvement than did any other initiative, about 20 percent of all 
expenditures. (See fig. 3.) 



Figure 2: Percentage of States that Reported Undertaking Nine Categories of 
Initiatives 

100 Percent 
90 




Catagories 

Source: GAO analysis of GAO survey data. 



^ese differences may be explained by the number of states undertaking the initiative or 
the amount of money Individual states allocated to a particular initiative, which in turn 
reflects the state’s size, available funds and priorities regarding child care quality. 
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Figure 3: States’ Reported Expenditures for Each Initiative, Fiscal Year 2000 



Resource & referral 

2% 

On-site caregiver training 
Incentives for accredilntion 



Satety equipment/improvement 
Olf-sile caregiver training 
Caregiver coinpcrisatiori 



All other activities 
Meeting state standards 

Enhanced inspections 



Note: The term fiscal year refers to the federal fiscal year. 

Source: GAO analysis of GAO survey data. 

States reported undertaking several initiatives to improve caregiver 
qualifications and compensation. Eighty-two percent of the states funded 
an off-site caregiver training initiative. One example is the Teacher 
Education and Compensation Helps (TEACH) program, which provides 
caregivers with scholarships to attend college classes related to child 
development. TEACH began in North Carolina and has been replicated in 
17 states. One-third of the states undertook initiatives to improve caregiver 
compensation through increased wages or benefits. For example, child 
care officials in Massachusetts use Quality Awards to reward child care 
staff and family child care providers with one-time bonuses for excellence 
in their work. A similar number of states reported funding on-site 
caregiver training, which provides caregivers with training and education 
opportunities at their place of employment. Officials in Wisconsin 
reported funding caregiver training for increased safety in family child 
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care homes. Taken together, these initiatives received about 25 percent of 
the expenditures states reported. 

Most states also reported undertaking initiatives to assist providers in 
meeting state standards and to reward providers for exceeding state 
standards. Thirty of the 42 responding states reported providing funding to 
assist child care providers in meeting state licensing standards, such as 
California’s provision of funding to assist child care providers with health 
and safety standards and a variety of training requirements. Additionally, 

29 of the 42 responding states reported funding safety improvements. For 
example. South Dakota’s health and safety funding offers child care 
providers up to 75 percent of the cost of safety equipment, such as 
windows designed to provide an escape route in the event of an 
emergency. Over half the states reported providing incentives for child 
care providers to become accredited or exceed state standards. Under its 
child care program, Wisconsin did so by setting the maximum 
reimbursement rate for providers that met accreditation standards, which 
exceed licensing standards, 10 percent higher than the regular 
reimbursement rate.® Initiatives to assist providers in meeting state 
standards received about 13 percent of states’ reported quality 
improvement expenditures. However, although many states reported 
funding safety improvements and offering incentives for accreditation or 
exceeding standards, these initiatives received the smallest shares of 
funding. 

Half the states reported initiatives devoted to enhancing inspections of 
child care facilities, either by increasing the frequency or the thoroughness 
of such inspections. Althou^ less commonly reported than several other 
initiatives, these inspection efforts received the second largest proportion 
of quality funds. 

Fi nall y, over half the states reported undertaking initiatives in the all other 
activities category. These included consumer education campaigns and 
improvement of the quality and availability of care for special populations, 
such as infants, toddlers, and children with special needs. California, for 
example, reported funding a variety of other activities including school- 
age curriculum and material development and a program for infant/toddler 
caregiver training coordinators. 



®See Background for a discussion of how standards for licensing and accreditation may 
vary. 
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The CCDF 4 percent set>aside funded nearly half of all state-reported 
expenditures on quality. (See fig. 4.) State funds were the next largest 
funding source, constituting almost one-third of all expenditures on quality 
improvement initiatives. States also madause of earmark funds, additional 
CCDF funds, and money available from TANF. Though some states did 
make use of funds available from private foimdations and other sources, 
this constituted a negligible proportion of the total. 



Figure 4; States’ Reported Expenditures from Each Funding Source, FIscai Year 
2000 




CCDF 4 percent set-aside 

6% 

TANF 

Additional CCDF 



CCDF earmarks 



State funds 



Source: GAO analysis of GAO survey data. 



State Officials Attributed 
Quality Improvement 
Decisions to Two Key 
Factors 



The views of state officials in both the executive and legislative branches 
of state government are considered in the allocation of federal and state 
child care funds for quality improvement. Officials in four of the five states 
we visited cited the views of state officials and previous research as two 
key factors in their selection of initiatives. In four states, decisions on how 
child care funds are allocated among the various quality initiatives are 
determined through the legislative process. For example, in 
Massachusetts, the state legislature’s perspective and previous research 
were cited as reasons that most quality initiative fimds were devoted to 
caregiver compensation. Officials we spoke with said the state legislature 
supported early literacy, which led officials to offer rate increases to 
providers that implemented early literacy initiatives. In addition, research 
led state officials to believe that improving caregiver compensation would 
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increase child care quality/ Therefore, these rate increases were meant to 
enable providers to improve caregiver compensation. (See fig. 5.) 



Figure 5: Massachusetts’s Expenditures on Quality Improvement Initiatives, Fiscal 
Year 2000 



Source; GAO analysis of GAO survey data. 

Officials in Termessee explained that several factors — recent events in the 
child care community, the state legislature’s perspective, and research — 
influenced the state’s emphasis on enhanced inspections. The Tennessee 
lead agency director told us that in 1999, the accidental deaths of two 
children in a child care van prompted the legislature to focus on quality 
improvement initiatives. Subsequently, Tennessee instituted a policy of 
crirninal background checks and an increase in the number of 
unannounced inspections of child care facilities. Tennessee’s distribution 
of funds emphasized this focus on inspections, as seen in figure 6. 
Tennessee now conducts six unannounced inspections of each facility per 




^In Who Cares for Americans Children, the National Research Council reviewed research 
showing that children, especially very young children, need enduring and consistent 
relationships with a caregiver. Yet, a significant number of caregivers at child care centers 
leave in a given year. Massachusetts’s recent study of caregiver recruitment and retention 
in the state confirmed the findings of other studies that caregivers who receive low wages 
are difficult to retain. 
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year.® In addition, Tennessee officials consulted research on child care 
quality to inform their decisions but did not sponsor evaluation, pointing 
to the trade-off between funding evaluations and direct services to 
improve quality. 



Figure 6: Tennessee's Fiscal Year 2000 Expenditures on Quality Improvement 
Initiatives 



1 % 

Meeting state standards 




Wisconsin officials told us that research and gubernatorial proposals 
influenced the selection of a range of quality improvement initiatives. State 
officials said they analyzed data on quality improvement programs, and 
consulted experts in the child care field. The state legislature and the 
governor also influenced priorities. For example, in January 1999, the 
governor put forth a proposal to direct $15 million into an Early Childhood 
Excellence Commission to develop high quality child care in low income 
neighborhoods. 



^Tennessee changed the licensing requirement from one unannounced inspection to six 
unannounced inspections per year and increased the licensing staff from about 80 to 159. 
Licensing staffs caseload is now about 35 facilities per full time staff person. Child care 
officials estimated they have spent about $6 million over 2 years on increased inspections, 
which are performed for all licensed providers. 
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In South Dakota, the decision to emphasize resource and referral agencies 
was guided by previous research and the governor’s perspective. State 
officials relied on existing child care quality research for making choices 
about how to improve quality because they believed that sponsoring 
evaluations would be too resource intensive. On the basis of previous 
research findings, state officials believed training caregivers to be the 
central mechanism through which child care quality could be improved. 
After obtaining the governor’s support, child care officials directed 
funding to resource and referral centers to train caregivers. (See fig. 7.) 



Figure 7: South Dakota’s Fiscal Year 2000 Expenditures on Quality Improvement 
Initiatives 



1 % 

Safety equipment/improvement 




By contrast in California, officials in the Department of Education, the 
CCDF lead state agency in that state, said that they have more discretion 
than other states in deciding which quality initiatives to fund. According to 
these officials, because the California Department of Education has its 
own constitutional officer, who is independent of the governor, California 
child care officials have more autonomy in their selection of initiatives. 
Department of Education officials explained that the department’s 
priorities — ^health and safety, best practices in early development and 
learning, and professional development — influenced the selection of a 
range of quality improvement initiatives. 




Page 18 



23 



6AO-02-897 Child Care Quality Improvement Initiatives 



Most States Spend More 
Than 4 Percent on Quality 
Improvement 



While HHS requires that states spend at least 4 percent of the CCDF on 
quality improvement, the mgjority of states reported expenditures for 
quality in excess of this minimum requirement in fiscal year 2000. In fact, 
in that year, 23 of 42 states reported expenditures representing 8 percent 
or more of the CCDF on quality related activities. We estimated that the 
percentage of the CCDF expended on quality ranged from 3 percent in 
California, Idaho and New Mexico to 38 percent in Kansas. (See table 4.) 

These reported expenditures are a snapshot of states’ expenditures for 
quality improvement in fiscal year 2000. Because of the distinct set of rules 
covering each of CCDF’s three funding streams, expenditures in that year 
by an individual state may have drawn on funds available from CCDF 
grants made in fiscal years 1998, 1999, or 2000. The percentage 
expenditure of funds from a particular fiscal year’s grant cannot be 
determined definitively until time limitations on all funding streams have 
expired." 



Table 4: States’ Reported CCDF Quality Improvement Expenditures In Fiscal Year 
2000 


State* 


State reported 
CCDF quality 
expenditures 


Average CCDF 
grant^ 


Percentage 
expended based 
on state report of 
expenditures'* 


Alabama 


$4,725,482 


$77,493,201 


6 


Alaska 


2.198,373 


23,327,524 


9 


Arizona 


6,197,000 


88,131,425 


7 


Arkansas 


1,998.221 


33,560,449 


6 


California 


20,700.000 


639,666,033 


3 


Colorado 


1,509.043 


39,379,887 


4 


Delaware 


871,830 


11,186,263 


8 


Georgia 


6,683,969 


131,333,851 


5 


Hawaii 


1,758,003 


20,180,147 


9 


Idaho 


967,425 


34,699,953 


3 


Illinois 


22,500,000 


211,895,440 


11 


Kansas 


14,315,739 


37,897,643 


38 


Kentucky 


2,670,451 


74,719.865 


4 


Louisiana 


6,349,109 


109,582,724 


6 


Maine 


4,080,000 


17,754,746 


23 


Maryland 


29,011,806 


101,462,889 


29 


Massachusetts 


15,498,039 


171,959,431 


9 











"See Background for a description of the funding stream rules. 
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State* 


State reported 
CCDF quality 
expenditures 


Average CCDF 
grant** 


Percentage 
expended based 
on state report of 
expenditures** 


Michigan 


14,662,330 


178,511,186 


8 


Minnesota 


8,124,224 


79,371,131 


10 


Mississippi 


1,770,041 


48,674,596 


4 


Missouri 


14,360,255 


84,871,476 


17 


Montana 


1,414,227 


18,137,917 


8 


Nebraska 


3,000,000 


25,634,209 


12 


New Hampshire 


1,169,031 


11,343,569 


10 


New Jersey 


10,700,000 


175,379,185 


6 


New Mexico 


1,119,790 


40,719,569 


3 


North Carolina 


9,520,719 


179,122,483 


5 


North Dakota 


1,404,790 


7,311,957 


19 


Ohio 


13,446,256 


170,661,715 


8 


Oklahoma 


3,500,000 


82,646,909 


4 


Oregon 


4,262,400 


41,411,987 


10 


Pennsylvania 


12,556,326 


193,953,622 


6 


South Carolina 


2,204,027 


44,374,421 


5 


South Dakota 


1,408,042 


9,727,797 


14 


Tennessee 


11,593,876 


120,436,809 


10 


Texas 


15,183,207 


288,255,772 


5 


Utah 


4,100,000 


33,632,252 


12 


Vermont 


1,613,691 


14,941,847 


11 


Virginia 


5,557,225 


91.906,307 


6 


Washington 


9,317,551 


162,038,398 


6 


Wisconsin 


18,500,000 


105,431,604 


18 


Wyoming 


1,241,670 


10,236,055 


12 


Average 


$7,470,575 


$94,591,882 


8 



*The following states did not reply to the questionnaire: Connecticut, Florida, Indiana, Iowa, Nevada, 
New York, Rhode Island, and West Virginia. The complete questionnaire submitted by the District of 
Columbia was not received in time to include the responses in our analyses. 



"In our survey of state CCDF lead agencies, states were asked to consider the amount of 4 percent 
set-aside funding available in fiscal year 2000, and to report the amount spent on quality in fiscal year 
2000. Funds available for expenditure may have included fiscal year 1999 or 2000 matching funds 
and fiscal years 1998-2000 mandatory and discretionary funds, including TANF transfers. With the 
exception of mandatory funds, all funds must be expended within three years. Because states’ 
expenditures in fiscal year 2000 could have drawn on grants made in fiscal years 1998-2000, the 
percentage of CCDF expended on quality was estimated by dividing the states’ response by the 
average of CCDF grant amounts in fiscal years 1998-2000. In cases where states included earmark 
or state funds-other than state funds used to match federal funds-in the response, these were 
removed before calculating a percentage. Thus, the numerator is based on states’ response to our 
survey of state CCDF lead agencies. We contacted state officials that reported unusually high or low 
expenditures to confirm their reports. The denominator is based on CCDF grant information for fiscal 
years 1998-2000 from HHS, which can be accessed at: 
http://www.acf.dhhs.gov/programs/ccb/research/00acf696/summary.htm and 
http://www.acf.dhhs.gov/programs/ccb/research/archive/99acf696/summary.htm. 
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Expenditures on Initiatives 
Directed to Providers 
Primarily Targeted Child 
Care Centers 



Among the 34 states that tracked the type of provider targeted, child care 
centers received over two-thirds of all expenditures for six initiatives that 
states targeted to individual providers. (See fig. 8.) However, nationwide, 
55 percent of all children whose care involves CCDF assistance are 
attending child care centers and 32 percent of all children are in center- 
based care. Thus, centers receive a larger share of quality improvement 
expenditures targeted to providers than the share of CCDF-subsidized 
children in their care. In addition, while there was insufficient information 
in states’ responses to analyze initiatives devoted to informal care, the 
policy research community has expressed interest in quality improvement 
initiatives targeted on these providers because we have the least 
information about them and a significant number of children are cared for 
in informal settings." 



‘“states were asked to estimate the proportion of quality improvement funds, including 
CCDF and all other funding sources, spent on different types of providers. This analysis 
refers only to the six initiatives for which funds are distributed to providers: caregiver 
compensation, on- and off-site training of caregivers, safety equipment and improvements, 
meeting state standards, and incentives for accreditation. Because these initiatives 
constitute 54 percent of all expenditures, this analysis eiccounts for just over half of all 
reported expenditures for quality improvement. Of the 42 states that responded to our 
survey, 34 were able to provide information about the type of provider targeted by one or 
more of the six initiatives that they funded. 

"“Informal care” refers to legally operating care given by adults, including friends and 
relatives, and is usually unregulated. 
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Figure 8: States’ Reported Expenditures on Quality Improvement Initiatives 
Targeted to Providers 




Source: GAO analysis of GAO survey data 



3% 

After-school care 



Family child care 



Center-based care 



When we looked at expenditures on individual initiatives by the thirty-four 
states, we saw the same pattern of emphasis on centers. (See table 5.) For 
the six initiatives, centers received the mgjority of funds, followed by 
family child care. Moreover, for initiatives related to meeting standards, 
the proportion of expenditures devoted to centers was smaller than for 
other initiatives, but still greatly exceeded the proportion devoted to 
family child care and after-school care. 



Table 5: States’ Reported Expenditures Devoted to Each Provider Type, by Initiative 


Initiative 


Percentage 
to centers 


Percentage to 
family child care 


Percentage to 
after-school 


Incentives for accreditation 


87 


13 


0 


Caregiver Compensation 


76 


23 


1 


Safety equipment/ 
improvements 


75 


20 


5 


On-site caregiver training 


74 


21 


5 


Off-site caregiver training 


70 


28 


2 


Meeting state standards 


68 


28 


4 



Source: GAO analysis of GAO survey data. 



o 

ERIC 
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However, when we examined expenditures on initiatives by individual 
states, the proportion of expenditures on quality improvement activities 
devoted to each provider type varied. (See table 6.) For example, 
Minnesota, Mississippi, Tennessee, Texas and Washington reported 
devoting 90 percent or more of quality expenditures to centers, and 
Delaware, Hawaii, Michigan, North Dakota, and Oregon reported devoting 
less than one-third of quality expenditures to centers. 

These differences can be explained in part by state-to-state differences in 
the proportion of children receiving CCDF subsidies that attend each type 
of provider. For example, in Michigan, 19 percent of children receiving 
CCDF subsidies attend centers, while in Tennessee, 73 percent of children 
receiving CCDF subsidies attend centers. Given the variation in the 
proportion of subsidized children attending center-based care, it would be 
reasonable for Michigan to devote relatively less of its quality expenditures 
to centers and for Tennessee to devote relatively more of its quality 
expenditures to centers. Because the CCDF set-aside is intended to 
improve child care for all children, the law allows states flexibility in 
developing programs and policies, including quality improvement 
initiatives and the types of providers targeted. 



Table 6: Comparison of Quality Improvement Expenditures Distributed to Individual Providers that Were Devoted to Each 
Provider Type, with Percentage of CCDF-SubsIdIzed Children, by State 



State 


Percentage of 
quality 
expenditures 
to centers 


Percentage of 
CCDF-subsidIzed 
children in centers 


Percentage 
of quality 
expenditures 
to FCCs 


Percentage of 
CCDF-subsidIzed 
children In FCCs 


Percentage of 
quality 
expenditures to 
after-school* 


Alaska 


79 


35 


15 


49 


6 


Arizona 


68 


73 


32 


14 


0 


Arkansas 


72 


82 


28 


18 


0 


California 


79 


71 


16 


17 


5 


Colorado 


47 


57 


48 


25 


5 


Delaware 


31 


55 


69 


35 


0 


Georgia 


70 


76 


28 


17 


2 


Hawaii 


27 


27 


64 


20 


9 


Kansas 


54 


36 


46 


50 


0 


Kentucky 


80 


61 


20 


29 


0 


Maine 


63 


29 


37 


33 


0 


Maryland 


57 


34 


43 


31 


0 


Massachusetts 


60 


56 


40 


23 


0 


Michigan 


18 


19 


69 


20 


13 


Minnesota 


100 


27 


0 


56 


0 


Mississippi 


100 


69 


0 


9 


0 


Missouri 


80 


37 


20 


42 


0 
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State 


Percentage of 
quality 
. expenditures 
to centers 


Percentage of 
CCDF-subsidized 
children in centers 


Percentage 
of quality 
expenditures 
to FCCs 


Percentage of 
CCDF-subsldlzed 
children In FCCs 


Percentage of 
quality 
expenditures to 
after-school* 


Montana 


39 


30 


61 


69 


0 


Nebraska 


33 


58 


67 


41 


0 


New Mexico 


62 


43 


33 


27 


5 


North Carolina 


83 


81 


15 


13 


2 


North Dakota 


28 


26 


72 


71 


1 


Oklahoma 


63 


81 


32 


19 


5 


Oregon 


0 


21 


100 


52 


0 


Pennsylvania 


59 


59 


28 


18 


13 


South Dakota 


47 


27 


50 


53 


3 


Tennessee 


90 


73 


5 


26 


4 


Texas 


95 


79 


5 


6 


0 


Utah 


87 


65 


13 


25 


0 


Vermont 


69 


44 


31 


50 


0 


Virginia 


84 


54 


16 


29 


0 


Washington 


100 


41 


0 


23 


0 


Wisconsin 


70 


60 


30 


39 


0 


Wyoming 


33 


31 


54 


40 


13 


Total percentage 


71 


b 


26 


b 


3 



“Information is not available on the percentage of CCDF-subsidized children in after-school care. 



‘’Average not applicable. 

Source: Percentage of quality expenditures devoted to centers, family child care and after-school 
care is based on GAO analysis of states’ responses to our survey of CCDF lead state agencies. 
Percentage of CCDF-subsidized children in centers and family child care Is based on data reported in 
U.S. House Of Representatives, Committee On Ways And Means, 2000 Green Book (Washington, 
D.C., 2000). 



Few States Have 
Evaluated the 
Effectiveness of State 
Quality Improvement 
Initiatives 



While few states have evaluated the effectiveness of state quality 
improvement iiutiatives on children’s development, some studies provide 
useful findings about them. Officials in fovur of five states we talked to 
explained that states must make trade-offs between serving more famihes 
and conducting evaluations of their own quality improvement initiatives. 
Out of a handful of state-sponsored studies, a few had study designs that 
isolated an iiutiative’s effect and survey response rates that provided 
reliable estimates. The research on child care quality does not evaluate 
iiutiatives as actually implemented by states, but a few studies, using 
rigorous methods, show theit some of the attributes of child care quality 
that these initiatives address, such as caregiver qualifications, affect 
children’s social, emotional and cognitive development. HHS has begun to 
support some analyses of states’ quality improvement efforts and could 
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play an even more important role in supporting rigorous studies of the 
initiatives states are imdertaking. 



Of the Handful of Studies 
on the Effectiveness of 
States’ Initiatives, Three 
Had Conclusive ladings 



Of the handful of studies that examined the effectiveness of states’ 
initiatives, three had methodological approaches sufficient to produce 
conclusive findings. In considering studies of the initiatives’ effectiveness, 
we looked primarily for studies that analyzed the effect of an initiative on 
children’s development. We also considered studies that examined effects 
on attributes of child care quality, such as caregiver qualifications or 
turnover. Improvements in attributes of child care quality can be seen as 
an intermediate step toward strengthening children’s development‘s One of 
the three studies with conclusive findings, sponsored by Florida, analyzed 
how Florida’s implementation of more stringent child-to-staff ratios and 
caregiver education requirements in child care centers was related to 
children’s cognitive and socio-emotional development over time. The two 
other studies with conclusive findings, sponsored by Massachusetts and 
Washington state, examined caregiver compensation and caregiver 
recruitment and retention rates. 



Taking measures of child care quality and children’s development before 
and after Florida instituted more stringent teacher-to-child ratios and 
caregiver education requirements, Florida’s study fovmd that a reduction 
in child-to-staff ratios and an increase in early education requirements for 
center providers contributed to gains in children’s development and the 
quality of early education and care they received. The study’s design 
allowed the contribution of child-to-staff ratios and caregivers’ education 
to children’s development to be examined, but, without a comparison 
group, was unable to isolate their effects completely. However, this 
linutation did not compromise the study’s findings. 

Massachusetts’s recruitment and retention study examined caregiver 
compensation, conducting a survey of providers regarding the reasons for 
the shortage and high turnover of providers in child care centers across 
Massachusetts. The study confirmed findings of other studies that 



‘^The methodological criterion we used was that to determine a program’s effect, an 
evaluation should employ an experimental or quasi-experimental design. See appendix I for 
a more detailed discussion of these study designs. In one case, however, we have included 
a study that used a nonexperimental design, but had very high quality data 

‘“C. Howes, E. Smith and E. Galinsky, The Florida Child Care Quality Improvement 
Study: 1996 Report (New York: Families and Work Institute, 1996). 
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caregivers who receive low wages are difficult to hire and retain. However, 
the study design it employed did not rule out explanations other than low 
salaries for the association between high turnover rates and workers 
receiving low wages. Washington State also evaluated caregiver 
compensation and retention, using a quasi-experimental design, but found 
no effect of the compensation on retention. 

It is important to acknowledge that while we looked at all of the studies 
we identified for evidence of the effectiveness of state initiatives, the 
studies that states sponsored may not always have been designed for that 
purpose and in some cases provided useful data on other issues that they 
were intended to address. For example, when the data sources used in 
nonexperimental studies meet data quality standards, as did data collected 
for the Massachusetts recruitment and retention study, state-sponsored 
studies can provide reliable information that is needed to address program 
design issues, such as setting reimbursement rates; to assess program 
implementation, such as examining the number of caregivers that have 
acquired training in child development; or to understand the child care 
market, such as determining the number of providers that offer health 
benefits to their caregivers.'^ Studies that collect this type of descriptive 
information also help in planning research that employs rigorous designs. 
We also recognize that more definitive studies are labor and resource 
intensive; studies that employ experimental designs are difficult and 
expensive to conduct. Similarly, surveys that involve low-income families, 
which may be needed for studies using quasi-experimental designs, require 
special procedures, such as the use of financial incentives or several 
rounds of follow-up with nonrespondents, to achieve a response rate that 
meets minimum data quality standards. Moreover, while state child care 
agencies may partner with universities or contract research organizations 
to conduct such studies as CCDF funding sources permit, officials in four 
of the five states we talked to explained that states must make trade-offs 
between serving more families and conducting evaluations of their own 
quality improvement initiatives. 

The remaining studies we identified did not meet our criteria for data 
quality, because of low survey response rates or self-selected samples. 
California conducted a comprehensive study of caregiver compensation. 



‘^See U.S. General Accounting Office, Early Childhood Programs: The Use of Impact 
Evaluations to Assess Program Effects, GAO-01-542 (Washington, D.C.: Apr. 16, 2001) for a 
more detailed discussion of other types and uses of program evaluation. 
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and Massachusetts conducted a second study of caregivers’ salaries, but 
both studies had very low response rates. North Carolina examined the 
reliability of criteria used in the state’s incentive for accreditation 
initiative, but the sample of providers they studied was self-selected and 
included few centers with low quality ratings. California also evaluated a 
statewide toU-free telephone line using administrative and survey data but 
this survey had a very low response rate. The results of our assessments of 
particular studies are described in greater detail in appendix 11. 



The Broader Literature 
Suggests, and a Few 
Studies Confirm, a Link 
between Child Care 
Quality Attributes and 
Children’s Developmental 
Progress 



The extensive body of research on child care quality that has been 
developed over the past 20 years has laid the foundation for understanding 
how the quality of care affects children’s progress. Child care research 
studied a variety of child care quality attributes and a few studies provided 
evidence of the effects of these attributes on children’s developmental 
progress. We examined reviews of the broad range of studies in this area 
to supplement the studies available on states’ initiatives. While the findings 
of this research suggest that some states’ initiatives are attempting to 
influence aspects of child care that have demonstrable effects on 
children’s development, this is not sufficient to determine that these 
initiatives are necessarily effective as implemented. 



In 2000, the National Research Council conducted a second 
methodological review of research on early childhood development that 
included research on child care quality.'® The council, like a team of 
reviewers sponsored by HHS’s Office of the Assistant Secretary for 
Planning and Evaluation and a team of reviewers sponsored by a 
foundation, examined the relationship between the structural attributes of 
quality and child-cai’egiver interactions on cliildren’s developmental 
progress. (See table 7.) The reviewers found that structural attributes, 
such as caregiver qualifications, child-to-staff ratios and smaller group 



‘®See Background for a description of the council’s first review. 

'^National Research Council and Institute of Medicine. Prom Neurons to Neighborhoods: 
The Science of Early Childhood Development. Conunittee on Integrating the Science of 
Early Childhood Development, edited by Jack P. Shonkoff and Deborah A. Phillips 
(Washington, D.C.: National Academy Press, 2000); Deborah Vandell and Barbara Wolfe, 
Child Care Quality: Does it Matter and Does it Need to be Improved? (Washington, D.C.: 
U.S. Department of Health and Human Services, Office of the Assistant Secretary for 
Planning and Evaluation, 2000); John M. Love, Peter Z. Schochet and Alicia L. Meckstroth, 
Are They in any Real Danger? What Research Does - and Doesn't — Tell us About Child 
Care Quality and Children's Well-Being, (Princeton, N.J.: Mathematica Policy Research, 
Inc., May 1996). 
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size, lead to developmental gains directly or fostered supportive and 
responsive caregiver behavior. All reviewers concluded that child- 
caregiver interactions that are responsive and supportive have positive 
effects on children's developmental progress. Each of these reviews also 
emphasized that studies of the effect of child care quality on child 
development should employ study designs and statistical methods that 
separate the effects of family characteristics on children's development 
from the quality of the child care setting.*'^ The third review in table 7 
includes a detailed discussion of the study designs and statistical methods, 
other than experimental design, that can be used to isolate the effect of 
child care quality. Among the large number of studies that were reviewed, 
the findings of those that met these criteria are summarized in appendix 
III. 



Table?: Major Reviewers’ Findings Regarding Child Care Quality Research 



Author and review 


Structural attributes that the review 
concluded contribute to children’s 
developmental progress or caregivers’ 
ability to create developmentally supportive 
environment 


Aspects of chlld-careglver Interactions 
that the review concluded contribute to 
children’s developmental progress 


Jack P. Shonkoff and Deborah A. 
Phillips, eds.. From Neurons to 
Neighborhoods 


Staff wages 
Lower staff turnover 
Caregiver education 
Caregiver training 


Caregiver continuity fosters the 
attachments that Improve social 
development 

Verbal environment that child care 
providers create contributes to children’s 
cognitive and language development 


Deborah Vandell and Barbara Wolfe, 
Child Care Quality: Does it Matter and 
Does it Need to be Improved? 


Smaller group size 
Lower child-to-staff ratios 
Caregiver education 
Careqiver training 


Emotionally supportive and cognitively 
enriching settings 


John M. Love, Peter Z. Schochet and 
Alicia L. Meckstroth, Are They in any 
Real Danger? What Research Does - 
and Doesn't— Tell us About Child Care 
Quality and Children's Well-Being 


Smaller group size 
Lower child-to-staff ratios 
Safer equipment and space 


Appropriate caregiving 
Developmentally appropriate practice 
Caregiver responsiveness 



methodological terms, when analyzing effects on children’s development, the need to 
separate the influence of family characteristics from the quality of the child care setting is 
called controlling for selection bias. Controlling for selection bias in conducting analyses of 
the effects of child care quality on children’s development and tying the size of an effect, 
when it can be determined, to the cost of achieving a change, are two key issues in 
research on child care quality and child outcomes. 




Page 28 



33 



6AO-02>897 Child Care Quality Improvement Initiatives 



These studies have shown relationships between structural attributes, 
child-caregiver interactions and children’s developmental progress that 
suggest many state initiatives are targeted on aspects of child care settings 
that have the potential for enhancing developmental outcomes. However, 
this is not sufficient to conclude that states’ initiatives are necessarily 
effective in enhancing child care quality. Such a conclusion would 
presume that they are not only targeted on aspects of child care quality 
with the potential to improve developmental outcomes, but that they are 
reaching providers in need of help and reflect the individual attributes and 
the context in which they were studied originally. For example, because 
many studies were conducted at an earlier time period, the qualifications 
of the caregivers studied may differ from the pool of caregivers available 
in the current labor market. In addition, the populations of providers that 
were drawn at the state or substate level are not necessarily similar to the 
populations of other states. Thus, while existing research findings help 
states plan their initiatives, rigorous evaluations of initiatives actually 
implemented by the states are needed to provide evidence of the 
initiatives’ effectiveness. 



HHS’s Role in Supporting 
Studies of States’ 
Initiatives 



Using CCDF funds set aside for research, demonstration and evaluation by 
the 2001 Consolidated Appropriations Act, HHS has developed a research 
agenda that includes studies of child care quality and a commitment to 
rigorous evaluation. HHS’s research agenda covers three goals and four 
categories of activity. The goals are (1) improve the capacity to respond to 
policy questions, (2) strengthen data collection and analysis systems for 
child care research, and (3) increase knowledge about the effectiveness of 
child care policies and programs on child development and in helping low- 
income families obtain and retain work.‘“ HHS supports these goals by 
funding state research partnerships, field-initiated research, 
demonstrations and evaluations and data collection and analysis systems 
for child care research. 



Of 23 quality-related on-going research projects HHS identified for us, 
components of three projects are investigating quality improvement 



‘®HHS’s Child Care Bureau, which administers the CCDF, oversees this research agenda In 
addition, HHS’s Office of Planning, Research and Evaluation (OPRE) is conducting the 
National Study of Child Care for Low-income Families. OPRE’s funding has come 
historicaUy under Section 1 10 of the Social Security Act. HHS’s Office of the Assistant 
Secretary for Planning and Evaluation focuses on crosscutting issues and filling g^s not 
covered by other HHS agencies, but has no dedicated budget for child care research. 
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initiatives. With funding in the state research partnership area, Minnesota 
and Massachusetts are examining how tiered reimbiusement strategies 
affect child care quality.*® Minnesota’s study is a component of a child care 
research partnership grant and Massachusetts’s is part of a grant that 
supports state data and research capacity building. Under the same grant, 
Massachusetts is evaluating the impact of caregiver compensation 
strategies on the quality of care. In addition, HHS has undertaken a 
multiyear evaluation of the implementation, net impact and benefits of 
selected state child care policies and strategies that will be conducted 
using experimental design to determine if there are effects. Currently in its 
first year, this 7-year, 9 million dollar study will examine state strategies in 
four locations. HHS has taken an important first step by initiating this 
evaluation. However, to represent the diversity of the 50 states and their 
quality improvement approaches, more research that employs 
experimental or quasi-experimental designs will be needed to determine 
the effectiveness of states’ quality improvement initiatives. 


Conclusion 


Few states have evaluated the effectiveness of their quality improvement 
initiatives. While current research provides states with promising 
directions in which to target their efforts, it offers little specific guidance 
on how to modify ongoing initiatives or the most cost effective placement 
of additional expenditures to improve quality. This limits states’ capacity 
to sustain and enhance initiatives that effectively improve the quality and 
availability of child care. Having additional rigorous research in this area 
would provide important information to both policymakers and 
administrators at all levels of government and support the Congress’ 
efforts to improve child care quality. 


Reconunendation 


We recommend that HHS include in its planned multiyear evaluation of the 
net impact and benefits of state child care policies, an analysis of the 
effects on children’s development of selected state quality improvement 
initiatives, such as off-site caregiver training or enhanced mspections. 


Agency Comments 


We obtained comments on a draft of this report from HHS’s 
Administration for Children and Families (ACF). These comments are 




*°“Tiered reimbursement” is reimbursement that offers higher rates to providers that meet 
certain quality standards set by the state. This is implemented under federal and state 
programs that subsidize care for low-income famihes. 
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reproduced in appendix IV. ACF also provided technical clarifications, 
which we incorporated when appropriate. 



HHS generally agreed with the findings of our report, expressing 
appreciation for the work we have done that makes the case for more 
research that evaluates the effectiveness of state quality improvement 
initiatives. HHS also described the department’s role in supporting studies 
of states’ initiatives and mentioned the technical assistance it provides 
states about relevant research findings through initiatives such as the 
National Child Care Information Center and the Healthy Child Care 
America campaign. 

In reference to our recommendation that HHS initiate research on state 
quality improvement initiatives, HHS expressed optimism that one or more 
of the analyses of state child care subsidy strategies in the multisite 
evaluation it is undertaking will test the effectiveness of state quality 
improvement initiatives. 



As requested, unless you publicly announce its contents earlier, we plan 
no further distribution of this report until 30 days after the date of this 
letter. At that time we will send copies of this report to the Secretary of 
Health and Human Services, appropriate congressional committees, and 
other interested parties. In addition, the report will be available at no 
charge on the GAO Web site at littp://www.gao.gov. If you or your staff 
have any questions about this report, please contact me on (202) 512-7215 
or Betty Ward-Zukerman, Assistant Director on (202) 512-2732. Other staff 
who contributed to this report are listed in appendix V. 




Mamie S. Shaul 

Director, Education, Workforce and 
Income Security Issues 
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We studied the initiatives that the states have implemented to improve 
child care quality. Our assessment examined two questions: (1) what 
initiatives have states undertaken with the 4 percent quality set-aside and 
other funding sources and (2) what evidence has been gathered, if any, 
about the effectiveness of states’ quality improvement initiatives? 


Scope 


The scope of our study was broader and more detailed than CCDF- 
mandated state reports because we asked states for data beyond what 
they reported to HHS. States receiving CCDF money must report to HHS 
aggregate expenditures to meet the set-aside requirement for child care 
quality improvement. However, states are not required to report to HHS 
how much they spend on specific quality improvement initiatives or which 
initiatives have shown evidence of effectiveness. Moreover, state or local 
governments may spend more money on quality improvements, using state 
funds or other resources, than is reflected in CCDF-mandated reports to 
HHS. We asked states to report all expenditures in federal fiscal year 2000 
by initiative, including those made with CCDF funds that may have been 
appropriated in prior fiscal years but spent in federal fiscal year 2000, plus 
funds firom other sources.' 


Methodology 


The CCDBG Act and HHS regulations give states discretion in deciding 
how to spend money to meet the 4 percent set-aside requirement. The set- 
aside may be spent on activities to provide comprehensive consumer 
education, parents’ choice of child care and on activities designed to 
improve the quality and availability of care children receive. 

The CCDBG Act defines quality in terms of activities states may undertake 
to meet the set-aside provision’s requirements. The act includes two 
provisions that apply to parents seeking child care: (1) comprehensive 
consumer education for parents and the public and (2) increased parental 
choice. The act identifies expenditures on these two activities as 
appropriate uses of set-aside funds but does not cite them as examples of 
child care quality and availability improvement activities. The provision 
does cite resource and referral services as an example of an activity 
designed to improve the quality and availability of care. 




‘Under CCDF provisions, states may spend money appropriated in a prior fiscal year in a 
later fiscal year. (See Background.) 
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Similarly, HHS regulations for the CCDF 4 percent set-aside aim at 
improving parents’ child care knowledge and choices and at improving the 
quality and availability of care children receive. The regulations include 
comprehensive consumer education and increasing parental choice as 
“quality” activities. The regulations state that activities to improve the 
quality of child care may include, but are not limited to, the following: 

• Improving resource and referral programs 

• Making grants or loans to providers to assist in meeting child care 
standards 

• Improving compliance with and enforcement of state and local licensing 
requirements 

• Providing training and technical assistance to providers in health and 
safety, nutrition, child abuse detection and prevention, and care of 
children with special needs 

• Improving salaries and compensation for staff who provide child care 
services. 

The regulations also include a provision that allows expenditures for any 
other activities that are consistent with the intent of the 4 percent set-aside 
section, which grants states considerable discretion. 

Because of the discretion that the law and HHS regulations allow states in 
selecting quality improvement initiatives, collecting information on quality 
improvement across all states required a common set of categories. The 
starting point for constructing categories for states’ quality improvement 
initiatives was the CCDBG Act and its regulations. However, the quality 
improvement activities specified in the law and regulations did not include 
all of the initiatives states are undertaking to improve the quality of care 
children receive. To ensure that our study analyzed all state spending for 
child care quality improvement, we developed nine categories for states’ 
child care quality improvement initiatives and asked states in which 
categories they funded activity, how much they spent, and the funding 
source. 

We developed the categories by combining federally designated activities 
with initiatives from contemporary child care quality analyses. To address 
the limitations of the federal activities, to create a more complete picture 
of state child care quality initiatives and to capture innovations in child 
care improvement, we added three categories to the federal categories. 
'Two were derived from quality improvement initiatives undertaken by the 
Department of Defense’s military child development program and one 
category was derived from literature analyzing child care quality 
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improvement. An “other quality-related activities” category was added 
because the regulations were not exhaustive and permitted states to 
develop initiatives not listed in regulations, provided that they were 
consistent with them. Thus, the category for other quality-related activities 
included initiatives that may be unique to a state and those that may foster 
the availability of high quality care, such as strategies that provide 
consumer education or increase parental choice. Because activities to 
provide comprehensive consumer education to parents and the public and 
increase parental choice were not included among the activities noted by 
the law or HHS regulations as designed to improve child care quality, we 
did not include these activities in our framework. However, states were 
free to report these or other quality-related activities when they construed 
them as such. Table 3 in the background section lists the nine initiative 
categories, their derivation, and descriptions. 



Survey Data Collection We surveyed CCDF lead state agency officials in the 50 states and the 

District of Columbia, asking that they report how much their state spent in 
each of the nine categories, the percentage of funds spent from each 
funding source in each category, types of providers and caregivers that 
initiatives targeted, and other information. For initiatives that included 
spending in more than one category, we asked state officials to record the 
spending for that initiative in the predominate category. When the draft 
data collection instrument was complete, GAO analysts and 
methodologists conducted a pretest in 4 states to ensure that the data 
collection instrument was clear and could be answered accurately in a 
reasonable amount of time. We made changes in the data collection 
instrument to incorporate comments from the pretest. Using a list of lead 
CCDF state agencies that was provided by HHS, we sent the data 
collection instrument on December 6, 2001, to 50 states and the District of 
Columbia by facsimile. 

The survey relied on state self-reporting of quality improvement iiutiatives 
and expenditures. While we did not independently verify states’ reports, 
we compared state survey responses to data collected firom our case study 
sites to provide some checks on the validity of state responses and cross- 
checked states’ estimates of 4 percent set-aside expenditures. We 
compared state expenditure data reported in the data collection 
instrument with expenditure data that states reported to HHS and resolved 
discrepancies through interviews with state officials. We worked with 
state officials to ensure a uniform understanding of the categories but the 
possibility exists that 2 states might have categorized similar iiutiatives 
differently. Forty-two of the 50 states and the District of Columbia 



Page 34 



6AO>02-897 Child Care Quality Improvement Initiatives 



Appendix I: Scope and Methodology 




responded to the survey, yielding a response rate of 82 percent. However, 
the District of Columbia’s complete data collection instrument was not 
received in time to be included in our analysis. 




For the analyses of how states’ devoted expenditures to providers of 
different types, we supplemented our survey data with data on CCDF- 
subsidized children from the House Committee on Ways and Means’s 
Greenbook. 


Case Studies 


We selected California, Massachusetts, South Dakota, Tennessee and 
Wisconsin as case study states. Our selection criteria included diversity in 
geography and population density; representation of a variety of child care 
quality improvement initiatives, such as direct and indirect attempts to 
improve caregiver compensation, initiatives directed at informal care 
giving; and whether a state used tiered reimbxusement rates as incentives 
for quality improvement. We also considered the population’s income 
distribution, licensing caseloads, use of Temporary Asistance for Needy 
Families funds, and whether state licensing requirements reflected NAEYC 
recommendations for child-to-staff ratios. We excluded states where we 
pretested our data collection instrument to minimize burden on any single 
state. 




The purpose of the case studies was to collect data that would explain or 
amplify data gathered by the data collection instrument. The case study 
protocol allowed state officials to provide explanations about what 
initiatives had been conducted, what factors influenced the state to 
undertake particular initiatives, what evaluations the state had performed, 
what innovations the state was undertaking, and whether the state had any 
unusual needs or problems in child care. 


Evidence of the 
Effectiveness of States’ 
Quality Improvement 
Initiatives 


Question 2 asked us to examine the evidence that had been gathered, if 
any, about the effectiveness of states’ quality improvement initiatives. We 
sought evidence of effectiveness in evaluations of the initiatives. We 
employed several search strategies to identify the evaluations. In our 
survey of CCDF lead state agencies, we asked states to identify 
evaluations they had conducted. We contacted HHS officials and child 
C 2 Lre reseeirchers regeirding their efforts to evaluate child caie quality and 
reviewed msgor reseeirch efforts. Our review included a discussion with 
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experts engaged in a study of child care quality funded by HHS’s Child 
Care Bureau.^ 

We also reviewed the literature on child care quality improvement 
initiatives, including information from previous GAO work, literature 
suggested by experts and information from electronic searches. Our 
review included both searching for studies on the effectiveness of quality 
improvement initiatives and conducting a citation search using a highly 
regarded state evaluation of child care quality improvement and an 
electronic search for reviews of research on child care quality. We also 
reviewed state reports regarding child care subsidies. 

From our survey and search strategies, we obtained reports from nine 
states that had sponsored research on quality improvement initiatives. We 
used a structured data collection instrument to analyze the reports. We 
collected information on the type of report, the report’s timeframe, quality 
improvement initiatives studied, design, data collection and analysis 
methods and findings. In addition, we assessed the study’s methodological 
strengths and limitations. 

Our assessment included both the quality of the data used in the 
evaluation and the methodological quality of the research. The criteria we 
used for assessing the data’s quality are shown in table 8. While we 
recognized that the administrative data were not collected to meet 
research standards, we paid particular attention to the administrative 
data’s completeness and the surveys’ response rates. When 30 percent or 
more of the administrative or survey data were missing, we looked for 
analyses showing no important difference between individuals 
represented in the data and those who were not included. 



^oni Porter et al.. Assessing the Child Care and Development Fund (CCDF) Investment 
in Child Care Quality: A Study of Selected State Initiatives (New York: Bank Street 
College of Education, 2002). 
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Table 8: Data Quality Criteria 


Survey data 


Administrative data 


Use of a random sample 


Correspondence to the entire study 
population 


Sample size greater than 30 


Sample size not applicable 


Response rate of 70 to 75 percent or 
greater 


High percentage of the study population for 
whom information was located in the data 
base 


Nonresponse analysis showing no 
important difference between individuals 
or families represented in the data and 
those missing from the data 


Comparative analyses showing no important 
difference between individuals or families 
represented in the data and those missing 
from the data if 30 percent or more of the 
records are missing 



Our assessment of the evaluations focused on the designs and analysis 
methods required to determine effects. The criteria we used in the 
assessment are shown in table 9. 



Table 9: Criteria for Assessing Evaluations 


Study component 


Criteria 


Design 


For an experimental design, selecting the group receiving the 
program and the control group randomly 
For both experimental and quasi-experimental designs, using a 
comparison group 


Data collection 


Meeting the criteria for survey and administrative data quality 
shown in table 18 


Data analysis 


Using a multivariate analysis procedure, as appropriate 




Using controls for influences other than the program 




Testing and correcting for iimitations such as nonrandom selection 
to the program and comparison group, and missing survey and 
administrative data 



An evaluation determines a program’s effect on its participants by isolating 
a program’s contribution from the effects of other influences that could 
have affected participant outcomes. To isolate the program’s influences, 
an evaluation studies two groups: those receiving program services and a 
similar group not receiving program services. Researchers compare the 
relevant outcomes of these two groups, such as children’s socioemotional 
development, to determine the program’s effect. 

The criteria for study design in table 9 apply to two types of evaluations: 
an experimental design and a quasi-experimental design. The two designs 



Page 37 



6AO-02-897 Child Care Quality Improvement Initiatives 



42 



Appendix 1: Scope and Methodology 



differ primarily in the way that the comparison groups are developed. In 
an experimental design, because comparison group members are selected 
randomly, researchers can compare outcomes to determine the program’s 
effect without using statistical controls for other factors that could have 
influenced the program. In a quasi-experimental design, the comparison 
group is composed of individuals who share characteristics with program 
participants, but who have not been randomly selected and who have not 
received program services.^ With this design, statistical controls, such as 
those provided by a multivariate analysis procedure, are needed to isolate 
the program from other factors that could influence outcomes. While there 
can be substantial practical difficulties in implementing experimental 
designs of social programs because program staff may be reluctant to 
participate and because of the tendency for comparison group participants 
to leave the study, there is no substantial debate about the desirability of a 
comparison group of some type in drawing conclusions about program 
effects. 

The criteria for assessing the administrative and survey data used in the 
evaluations were the same data quality criteria we discussed above. The 
criteria for data analysis in table 9 refer to the need to control for factors 
other than the program when program participants and comparison group 
members are not randomly selected. They also encompass additional 
analyses that may be needed if the group receiving program services and 
the comparison group were not randomly selected or to determine if 
nfissing data affect the reliability of the estimates of the program’s effect. 

Finally, several of the studies we assessed and the reviews of child care 
research we examined made reference to scales for measuring child care 
quality. In child care quality research, the structural attributes of quality 
are measured directly by, for example, counting the number of children 
per caregiver in classrooms or the years of education that a caregiver has 
attained. Howev6r, because cluld-caregiver interactions must be observed 
and recorded for research purposes, researchers have developed various 
scales to measure it. These scales contain numerous items that evaluate 
the areas of personal care routines, furnishings, language reasoning 
experiences, motor activities, creative activities, social development, and 
staff needs. 'Three of the most well known scales used in measuring 
process quality are the Early Care Environment Rating Scale (ECERS), the 
Infant/Toddler Environment Rating Scale (ITERS), and the Family Day 



“See GAOOl-5'12 for a detailed description of experimental and quasi-experimental designs. 
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Care Rating Scale (FDCRS).'' The ECERS and ITERS scales measure child- 
caregiver interactions at center-based care while FDCRS measures 
process quality in child care homes. 

One of the reviews assessed the scales’ strengths and limitations. The 
strengths of these three scales are their ease of use, reliability, good 
psychological measurement properties, and ability for cross-study 
comparisons. However, the reviewer pointed out, these scales also have 
some limitations. For example, their global composite scores combine 
features of various environments and influences when some of these areas 
may have greater influence on children’s development as compared to 
others. Additionally, these scales are setting-specific which means it is not 
possible to make simple comparisons across types of care or to combine 
scores in meta analyses. Another review pointed out that none of the 
existing scales include measures for the aspects of informal care that 
parents see as important, including such characteristics as shared values 
and language, a homelike atmosphere, the opportunity for a child to be 
cared for with siblings, and flexibility about hours and schedule. 



■‘ECERS and FDCRS were developed by Harms and Clifford, 1980, and ITERS was 
developed by Harms, Cryer, and Clifford, 1990. 
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Improvement 










Table 10 presents studies that we identified as attempting to examine the 
effectiveness of state quality improvement initiatives. They are ordered so 
that studies that meet GAO’s criteria for data and research quality are 
presented first. The table provides the study’s title, the quality 
improvement initiative the study examined, mqjor findings, and 
methodological strengths and limitations. Studies in the table were 
conducted or sponsored by the state. 


Table 10; State-Initiated Studies of Quality Improvement 


Study and quality 

Improvement 

initiative 


Purpose 




Findings 


Strengths 


Limitations 


The Florida Child 
Care Quality 
Improvement Study: 
1 996 Report 

• Classroom ratios 

• Off-site training 


To determine the effects 
of Florida’s new ratio and 
education requirements 
on children’s cognitive and 
socio-emotional 
development. 


Reduced child to staff ratios 
significantly contributed to gains 
in children’s cognitive and 
language development and 
attachment to their teachers. 


Before and after study 
design analyzed effects 
on children’s 
development 

Random sample 

Representative sample 

Multivariate analyses 
conducted 


28 percent of the 
child care centers in 
the original sample 
had to be replaced 
at the second 
measurement time 

Results not 
generalizable 
beyond four 
counties 












No comparison 
qroup 


Massachusetts 
Recruitment and 
Retention Study 

• Caregiver 
compensation 


To determine the reasons 
for the shortage and high 
turnover of child care 
center providers. 


Low wages are associated with 
difficulty in recruiting and 
retaining child care center staff. 


Stratified random sample 

100 percent response 
rate for telephone survey 
of center directors 

Large sample size of 
center directors 


Nonexperimentai 

design 
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Study and quality 
improvement 
initiative 

Washington State 
Child Care Career 
and Wage Ladder 
Pilot Project 

• Caregiver 
compensation 

• Retention 

• Off-site training 



Purpose 

To determine the effects 
of a career and wage 
ladder pilot project, which 
establishes specific job 
titles and related wages 
based on teacher 
education and experience, 
on staff retention, 
education, wage, and 
benefit changes in child 
care centers. 



Validating North 
Carolina’s 5-Star 
Child Care Licensing 
System 



To determine if the state 
licensing system 
accurately portrays the 
quality of child care 
centers. 



• Incentives for 
accreditation 



Massachusetts Child 
Care Center & 
School Age Program 
Salary and Benefits 
Report 



To determine the starting 
salary ranges and benefits 
for different types of child 
care center staff. 



• Caregiver 
compensation 

California Quality 
Improvement 
Program Evaluation: 
Healthline 

• Children’s health 
and safety issues 



To determine whether 
Healthline, toll-free 
telephone line for 
information on children’s 
health and safety, 
effectively reached 
providers and parents and 
promoted child health and 
safety. 



Findin gs 

A statistically significant 
difference was not found 
between pilot and comparison 
centers on retention rates and 
average length of employment 



Stren g ths 

Quasi-experimental 
study design with a 
comparison group 

High survey response 
rates 



Limitations 

Measuring progress 
toward the goal of 
improved quality of 
child care was 
beyond the scope of 
this phase of the 
evaluation 



Stratified random sample 

Large sample size for 
mail surveys sent to pilot 
and comparison center 
directors 



Licensing system accurately 
reflects the overall quality of a 
child care center. Centers with 
different ratings exhibit 
meaningful differences. 



Staff in licensed group child care 
centers started at $7 to $17 per 
hour. 

Staff in licensed school age child 
care programs started at $6.50 to 
$17.30 per hour. 



Comparison of ECERS 
scores, determined by 
an independent team of 
university-based 
researchers and 5-star 
ratings constituted an 
independent validation of 
the 5-star assessment 
Data gathered from a 
variety of populations 
which provided a more 
representative picture of 
recruitment and retention 



Small sample size 
of directors 
interviewed by 
telephone 

Sample of 
telephone- 
interviewed 
directors was 
judgmentally 

selected 

Nonexperimental 

design 

Participants self- 
selected 



Nonexperimental 

design 

Low survey 
response rate 



79 percent of Healthline calls 
were providers 



Overall, callers matched 
distribution of state population 



Use of both 
administrative and 
survey data to increase 
population coverage 

Use of multiple 
measures of Healthline’s 
outreach 



25 percent of providers statewide 
had heard of the Healthline. 



Nonexperimental 

design 

Callers’ county used 
to represent callers’ 
characteristics 

Response rates 
below standard 



Most callers reported that 
information they received met 
their needs. 



Nonresponse 
analysis not 
conducted 
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Study and quality 
improvement 
initiative 

California Child Care 
and Development 
Compensation 
Study: Towards 
Promising Policy and 
Practice 

• Caregiver 
compensation 



Purpose 

To determine to what 
extent the educational 
status of child care 
providers affects their 
wages, benefits, and 
turnover in different types 
of centers. 



Findings 

Approximately one third of 
caregivers hold a B.A. or higher, 
with no statistically significant 
differences among center types. 

Teachers in public centers 
earned $1.55 more per hour than 
teachers in nonprofit centers and 
$2.10 per hour more than 
teachers in for-profit centers. 

Child care staff received total 
benefits valued at 23 percent of 
for-profit wages and 30 percent 
wages at nonprofit and public 
centers. 



Stren g ths 

Quasi-experimental 
study design with a 
comparison group. 

Random samples. 

Standard tests of 
significance. 

Multivariate regression 
to analyze effects 

Findings compared to 
analogous study results 
to compensate for data 
limitations. 



Caregiver turnover in public 
centers lower than for profit 
centers. Caregivers in centers 
with the highest turnover had the 
lowest wages. 



Limitations 

Center survey 
response rate of 45 
percent was below 
standard 
For-profit center 
response rate of 20 
percent was below 
the standard 

No nonresponse 
analysis or sample 
weights to adjust for 
low survey 
response 

Less stringent tests 
of significance for 
analyses of effect 



The Colorado 
Expanding Quality 
Infant and Toddler 


To determine the effect of 
a 45-hour infant and 
toddler training curriculum 


Of the classrooms where a child 
care provider had participated in 
the training, 97% achieved 


Random sample 
Before and after study 


Nonexperimental 

design 


Care Initiative 


on the quality of care 
provided by students 


quality scores. 


design 


Survey response 
rate of 43 percent 


• Off-site training 


participating in the 
training. 


99% of the students who 
completed the post-training 
assessment survey reported the 
training would help them improve 
the quality of care they provided. 

All of the training instructors 
reported they felt the quality of 
care their students provided 
improved as a result of the 
traininq. 


Tests of significance 
conducted 


was below standard 

Nonresponse 
analysis was not 
conducted 
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Study and quality 
improvement 

Initiative 

Smart Start and 
Child Care in North 
Carolina: Effects on 
Quality and Changes 
over Time 

• Caregiver 
compensation 

« On-site training 

• Incentives for 
accreditation 



Purpose 

To determine the effect of 
Smart Start activities 
(enhanced subsidies for 
higher child care quality or 
higher teacher education; 
license upgrades; 
technical assistance; 
quality improvement and 
facility grants; teacher 
education scholarships; 
and teacher salary 
supplements) on the 
quality of child care over 
time. 



Finding s 

Quality of child care increased 
significantly from 1994 to 1999, 
with a greater increase from 
1 994 to 1 996 and a smaller 
increase from 1996 to 1999. 
Twice as many centers in 1999 
compared to 1994 scored In the 
“good to excellent’ quality range. 

Extensive previous participation 
in Smart Start does not 
guarantee that a center’s current 
quality is high. 



Strengths 

High response rates in 
1 994 and 1 999 samples 

Longitudinal design 

Multiple regression 
analyses 

Large sample sizes 

Tests of significance 
conducted 



Number of teachers participating 
in programs to obtain more 
education increased. 



Number of teachers with some 
college coursework increased. 



Limitations 

Nonexperimental 

design 

Smaller sample in 
1999 as compared 
to 1994 and 1996 
(attrition of 52 
centers, or 28 
percent less) 

Did not correct for 
selection effects 
among centers 
participating in 
Smart Start 

Low response rate 
in 1996 (68 percent) 



Percentage of centers licensed 
at higher levels and percentage 
of nationally accredited centers 
increased. 



Benefit levels were positively 
related to participation in Smart 
Start. 



Median teacher turnover 
remained steady at 17-20 
percent. 

Group sizes and teacher-child 
ratios have remained fairly 
constant 
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Study and quality 

improvement 

initiative 


Purpose 


Findings 


Strengths 


Limitations 


Oklahoma Tiered 
Licensing and 
Differential Quality 
Study 

• Incentives for 
accreditation 

• Off-site training 


To examine the 
variability in child 
care centers 
operating within 
the different 
regulatory 
climates. 


Accredited centers, whether two-star or 
not, were more likely than licensed and 
two-star by criteria centers to offer better 
quality child care. 

Centers with a smaller proportion of their 
enrollment receiving subsidies were 
more likely to offer better quality care. 


Used multiple measures 
to evaluate the quality of 
care 

Tests of significance 
conducted 


Nonexperimental 

design 

Nonrandom sample 

Sample limited to 
certain age groups 
of children 






Master teachers who qualified by 
education were more likely to offer better 
quality care. 
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Appendix III: Child Care Quality Research 
Findings 



Appendix III presents findings from two of the reviews of research on 
child care quality, discussed in the letter of the report, that found effects of 
the structural attributes of quality 2ind child-caregiver interactions on 
children’s developmental progress.' These two reviews provided sufficient 
methodological detail about the studies they assessed to identify those 
that met the criteria for 2inalyses of the effects of child care quality that we 
describe in appendix I. The findings from these reviews are broken out in 
appendix III by those that are linked to children’s socio-emotional 
development, cognitive development, 2ind development over time. We 
present only findings of studies the reviewers examined that could isolate 
the effect of child care quality on children’s development. 

The attributes that underlie the quality improvement initiatives being 
implemented by the states are primarily structural. These include child- 
teacher ratios, group class size, caregiver formal education, caregiver 
specialized training, classroom structure, 2ind health 2ind safety features. 
While research shows that child-caregiver interactions are equally as 
important in improving the quality of child care, states’ initiatives tend to 
address these attributes only through such initiatives as incentives for 
achieving accreditation.^ Thus, findings from research examining 
structure attributes may be more useful for targeting states’ quality 
improvement initiatives. 



‘John M. Love, Peter Z. Schochet, and Alicia L. Meckstroth, Are They in Any Real Danger? 
What Research Does — and Doesn’t — Tell Us about Child Care Quality and Children’s 
Well-Being, (Princeton, N J.: Mathematica Policy Research, Inc.); Deborah Lowe Vandell 
and Barbara Wolfe, Child Care Quality: Does it Matter and Does it Need to be Improved? 
(Washington, D.C.: U.S. Department of Health and Human Services, Office of the Assistant 
Secretary for Planning and Evaluation, 2000). 

^Love, Schochet and Meckstroth, Danger; Vandell and Wolfe, Child Care Quality. 
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Findings 



Child Care Quality 
Linked to Socio- 
Emotional 
Development 



Several studies have determined that children cared for in high quality 
child care settings show positive socio-emotional development. Lower 
child-to-adult ratios and smaller class sizes improve children’s social and 
emotional development. Lower child-to-adult ratios result in children 
appearing less apathetic and distressed;^ fewer behavior problems at 24 
and 36 months of age;^ enhancements in children’s social development;^ 
and teachers and children interacting more beneficially.® Smaller class 
size has been linked with children being more cooperative and less hostile 
and conflict-prone in their interactions with others;^ fewer behavior 
problems at 24 and 36 months of age;® and enhancements in children’s 
social development.® Additionally, researchers have found that when 
caregivers have more formal education and specialized training, children 
are more cooperative,^® have fewer behavior problems at 24 and 36 months 
of age," and have a greater security of attachment.*^ Low staff turnover is 
associated with children being more competent in social development, and 
higher staff wages are linked with higher-quality centers.*® 



^R. Ruopp, J. Travers, F. Glantz, and C. Coelen, Children at the Center: Final Report of the 
National Day Care Study (Cambridge: Abt Associates, 1979). 

'NICHD ECCRN, “Effect Sizes from the NICHD Study of Early Child Care,” paper 
presented at the Biennial Meeting of the Society for Research in Child Development, 
Albuquerque, N. Mex., April 1999. 

®Ruopp, Travers, Glantz and Coelen, Children at the Center. 

®Marcy Whitebook, Carollee Howes, and Deborah Phillips, Who Cares? Child Care 
Teachers and the polity of Care in America: Final Report: National Child Care 
Staffing Study (Berkeley: Child Care Employee Project, 1989). Reviewers indicated study 
controlled for family characteristics. Study design and analysis procedures were not 
identified. 

^Ruopp, Travers, Glantz and Coelen, Children at the Center. 

®NICHD ECCRN, “Effect Sizes.” 

®Ruopp, Travers, Glantz and Coelen, Children at the Center. 

*®Ruopp, Travers, Glantz and Coelen, Children at the Center. 

**NICHD ECCRN, “Effect Sizes.” 

*^Ellen Galinsky, Carollee Howes, and Susan Kontos, The Family Child Care Training 
Study: Highlights of Findings (New York: Families and Work Institute, 1995). 

*®Whitebook, Howes, and Phillips, Child Care Staffing Study. Reviewers indicated study 
controlled for family characteristics. Study design and analysis procedures were not 
identified. 
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Finally, children appeared happier and more positively engaged with their 
classmates when their caregivers were more involved, positive, and 
responsive with them.‘^ Children showed greater interest and participation 
when centers had ECERS scores in the high-quality range.'® 



Child Care Quality 
Linked to Cognitive 
Development 



Child care quality research also has found that high quality care 
contributes to improvement in children’s cognitive development. Lower 
child-to-staff ratios are linked with enhancements in children’s cognitive 
development/® including improvements in general knowledge, receptivity 
to language/^ and at 36 months, school readiness, and language 
comprehension scores/® Smaller groups are associated with enhancements 
in children’s cognitive development,*® school readiness, and language 
comprehension scores/® 

Caregiver education and training are also associated with better cognitive 
development in children. More highly educated or trained caregivers have 
been found to improve children’s school readiness and language 
comprehension scores.^* In addition, low staff turnover is associated with 
children being more competent in language development.^^ 



*^L.L. Hestenes, S. Kontos, and Y. Bryan, “Children’s Emotional Expression in Child Care 
Centers Vaiying in Quality,” Early ChiMkood Research Quarterly 8 (1993): 295-307; S.D. 
Holloway and M. Reichhart-Erickson, “The Relationship of Day Care Quality to Children’s 
Free Play Behavior and Social Problem-Solving Skills,” Early Childhood Research 
Quarterly 3(1988): 39-53; S. Kontos and A. Wilcox-Herzog, “Influences on Children’s 
Competence in Early Chil^ood Classrooms,” Early Childhood Research Quarterly 12 
(1997): 247-262. 

‘®E.S. Peisner-Feinbeig and M. R. Burchinal, “Relations between Preschool Children’s Child- 
Care Experiences and Concurrent Development: The Cost, Quality, and Outcomes Study,” 
Merrill — Palmer Quarterly 43 (1997): 451-477. 

*®Ruopp, Travers, Glantz and Coelen, Children at the Center. 

*^Ruopp, Travers, Glantz and Coelen, Children at the Center. 

*®NICHD ECCRN, “Effect Sizes.” 

*®Ruopp, Travers, Glantz and Coelen, Children at the Center. 

^®NICHD ECCRN, “Effect Sizes.” 

^*NICHD ECCRN, “Effect Sizes.” 

^^Whitebook, Howes, and Phillips, Child Care Staffing Study. Reviewers indicated study 
controlled for family characteristics. Study design and analysis procedures were not 
identified. 
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The experiences that occur in the environment in which children are cared 
for are also linked to cognitive development. Higher quality experiences 
are associated with children performing better on tests of language, “ 
intelligence,^^ and reading.^ In addition, chUd-to-caregiver interactions are 
linked to better cognitive development and improvements in cognitive 
competence during free play after participating in activities involving art, 
playing blocks, and dramatic play.“ Caregiver language stimulation (in 
both centers and homes) is associated with better performance on 
standardized language tests.^’ 



Studies that examined children’s development over time have shown that 
high quality care is a predictor of improvement in children’s receptive 
language and functional communication skills, verbal IQ skills, cognitive 
skills, behavioral skills, and attainment of higher math and receptive 
language scores. Changes in these skills can be detected with greater 
certainty when examined over time. 

When children attend classrooms that meet recommended cluld-to-staff 
ratio guidelines, they exhibit better receptive language and communication 



Child Care Quality 
Linked to Child 
Development Over 
Time 



Dunn, S. A. Beach, and S. Kontos, “Quality of the Literacy Environment in Day Care and 
Children’s Development,” Journal of Research in Childhood Education 9 (1994): 24-34; H. 
Goelman, “The Relationship between Structure and Process Variables in Home and Day 
Care Settings on Children’s Language Development,” in The Practice of Ecological 
Research: From Concepts to Methodology, edited by A. Pence and H. Goelman ( N.p., 

1988); K McCartney, “Effect of Quality of Day-Care Environment on Children’s Language 
Development,” Develojymental Psychology 20 (1984): 244-260; NICHD ECCRN, “The 
Relation of Child Care to Cognitive and Language Development,” in Child Development (in 
press); Peisner-Feinberg and Burchinal, “Preschool Children’s Child-Care Experiences”; E. 
Schliecker, D. R. White, and E. Jacobs, “The Role of Day Care Quality in the Prediction of 
Children’s Vocabulary,” Canadian Journal of Behavioural Science 23 (1991): 12-24. 

^L. Dunn, “Proximal and Distal Features of Day Care Quality and Children’s Development,” 
Early Childhood Research Quarterly 8 (1993): 167-192. 

^^Peisner-Feinberg and Burchinal, “Preschool Children’s Child-Care Experiences.” 

S. Kontos and A. Wilcox-Herzog, “Influences on Children’s Competence in Early 
Childhood Classrooms,” Early Childhood Research Quarterly 12 (1997): 247-262. 

^^Dunn, Beach, and Kontos, “Quality of the Literacy Environment in Day Care”; Goelman, 
“The Relationship between Structure and Process Variables ”; McCartney “Effect of Quality 
of Day-Care;” NICHD ECCRN, “The Relation of Child Care”; Peisner-Feinberg and 
Burchinal, “Preschool Children’s Child-Care Experiences”; Schliecker, White, and Jacobs, 
“The Role of Day Care Quality.” 



o 
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skills over time.“ However, when children attend classrooms with higher 
than recommended child-to-staff ratios, the children, once they reach 
preschool and kindergarten, are rated by their teachers as being more 
difficult and hostile. In addition, these children tend to engage in less 
social play and display less positive emotions.^ Caregivers with more 
specialized training were associated with children having higher math and 
receptive language scores over time.™ Girls whose caregivers had at least 
14 years of education displayed better cognitive and receptive language 
skills over time.'" On the other hand, once in preschool and kindergarten, 
children who (during their first 3 years of age) attended child care where 
caregivers had no formal child development training or where they were 
cared for by more than two primary caregivers in a year, were rated by 
their teachers as being more difficult and hostile. In addition, those 
children engaged in less social play and displayed less positive emotions.™ 



“M.R. Burchinal, J.E. Roberts, R. Riggins, S. A. Zeisel, E. Neebe, and D. Biyant, “Relating 
Quality of Center Child Care to Early Cognitive and l^guage Development 
Longitudinally,” in Child Development (in press). 

^*^C. Howes, “Can the Age of Entry into Child Care and the Quality of Child Care Predict 
A(^ustment in Kindergarten?," Developmental Psychology 26 (1990): 292-303. 

”D.M. Blau, “The Effects of Child Care Characteristics on Child Development," Jourml of 
Human Resources 34, no. 4 (1999): 786-822. 

Burchinal et al., “Relating Quality of Center Child Care.” 

“Howes, “A(^justment in Kindergarten." 
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Finally, when more involved and invested caregivers care for children 
during their first three years, kindergarten teachers report that those 
children have fewer behavior problems and better verbal IQ scores In 
addition, higher quality experiences are associated with children 
exhibiting better receptivity to language and communication skills over 
tirne^ 



“Howes, “Adjustment in Kindergarten.” 

“Burchinal et al., “Relating Quality of Center Child Care.” 
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Appendix IV: Comments from the 
Department of Health and Human Services 




DEPARTMENT OF HEALTH AND HUMAN SERVICES 



Administration for Children ano Families 
Office of the Assistant Secretary. Suite 600 
370 LEnfant Promenade, S.W. 

Washington. D.C. 20447 



AUG 2 3 2UC2 



Ms. Mamie S. Shaul 
Director 

Education, Workforce, and Income Security Issues 
U.S. General Accounting Office 
Washington, D.C. 20548 

Dear Ms. Shaul, 

I hank you for the opportunity to comment on the General Accounting Office draft report 
entitled: " Child Care: States Have Undertaken a Variety of Quality Improvement Initiatives, 
hut More Evaluations of Effectiveness Are Needed" GAO-Q2-897 . 

Attached are the Administration for Children and Families' comments. If you have any questions 
regarding our comments, please call Shannon Christian, Associate Commissioner, Child Care 
Bureau, at (202) 690-6782. 



Sincerely, 




Wade F. Morn; Ph.D. 
Assistant Secretary 



for Children and Families 



Enclosure 
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COMMENTS OF THE ADMINISTRATION FOR CHILDREN AND FAMILIES ON THE 
GAO DRAFT REPORT: "CHILDCARE: STATES HAVE UNDERTAKEN A VARIETY OF 
QUALITY IMPROVEMENT INITIATIVES. BUT MORE EVALUATIONS OF 
EFFECTIVENESS ARE NEEDED” GAO-02-897 

The Administration for Children and Families (ACF) appreciates the opportunity to comment on 
this report, which addresses an important topic. 

GAO Recommendations 

GAO recommends that HHS include in its planned multi-year evaluation of the net impact and 
benefits of state child care policies, an analysis of the effects on children’s development of 
selected state quality improvement initiatives, such as off-site caregiver training or enhanced 
inspection. 

ACF Comments 

The Child Care and Development Fund (CCDF) provides states with significant flexibility in 
determining where to target their quality set-aside funds. As GAO indicates, these decisions 
often involve governors and state legislatures. While most states use some of their CCDF funds 
for new and innovative quality initiatives, they also use CCDF for basic services such as resource 
and referral and ensuring the health and safety of children in care. This is consistent with the 
GAO finding that 35 percent of reported state expenditures are for enhanced inspections and to 
assist providers in meeting standards and making safety improvements. 

ACF, through its Child Care Bureau, provides technical assistance and support to assist states in 
responding to the competing demands associated with improving child care affordability and 
supply while making quality investments. Through written materials, meetings of state child 
care administrators, and technical assistance efforts such as the National Child Care Information 
Center (NCClC) and Healthy Child Care America, the Bureau responds to issues of concern to 
states and provides information about innovative practices and relevant research findings. For 
example, through NCCIC, the Bureau has produced publications regarding the recruitment and 
retention of child care staff and state initiatives that provide rate incentives for higher quality 
care. On-site training and technical assistance have been provided to states regarding strategies 
to improve the quality of legally exempt care. 

Through the Bureau's technical assistance efforts, it has become clear that states have an 
increasing interest in scientifically-based evidence on which to support decisions about 
improving child care services and systems. In particular, states indicate the desire for research 
that can help them make the best investments from among possible quality initiatives. 

ACF has made substantial commitment in research designed to promote better understanding of 
state and local child care policies and child care markets and the choices low-income parents 
make within the context of those policies and markets. Much of this work, such as the National 
Study of Child Care for Low-Income Families, has been descriptive in nature and complements 
studies sponsored by the National Institutes of Health and the Department of Education. 
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In addition, the Child Care Bureau has embarked on an ambitious effort to build the capacity for 
child care research at national, state and community levels. 'Fhis effort includes a Research 
Consortium comprised of nearly 50 current and prior-year grantees and contractors who have 
been funded by the Bureau to conduct policy-relevant research on child care issues. Through the 
Research Consortium, researchers come together to address critical definitional, policy, and 
methodological issues. This work provides the basis for sound, more rigorous research on child 
care issues. 

Finally, two research priorities have been initiated to encourage state and community-level 
research efforts. First, competitive grants are being offered to state CCDF lead agencies to assist 
them in improving their capacity to conduct policy-relevant research and analysis. This effort is 
intended to further the design and implementation of cost-effective child care policies and 
programs that are based on solid evidence. 

Second, as noted in the GAO study, ACF has contracted for a multi-site evaluation to study the 
impact, implementation, and costs and benefits of selected child care subsidy strategies. For 
purposes of this study, "subsidy strategies" are defined broadly to include policies that govern 
state child care programs and state initiatives to improve child care quality and outcomes for 
families and children. This evaluation will expand knowledge about child care by investigating 
causality using experimental methods. Exploratory work is in progress with states to identify 
sites and topics that may be appropriate for study using experimental methods. Early indications 
are that states are interested in knowing more about how they can best support child care quality, 
school readiness, and early literacy. We are optimistic that one or more of these studies will test 
the effectiveness of state quality initiatives. 

We appreciate the work your office has done to highlight state efforts to improve child care 
quality and to make the case for more research that evaluates the effectiveness of these efforts. 
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